Fluid user interface such as immersive multimediator or input/output device with one or more spray jets

ABSTRACT

A fluid user interface is presented for applications such as immersive multimedia. In one embodiment, one or more sprays or jets create an immersive multimedia environment in which a participant can interact within the immersive multimedia environment by blocking, partially blocking, diverting, or otherwise engaging with a fluid, to create computational input. When the fluid is air, a keyboard can be implemented on cusions of air coming out of various nozzles or jets. When the fluid is water, the invention may be used in environments such as showers, baths, hot tubs, waterplay areas, gardens, and the like to create a fun, playful, or wet user-interface. In some embodiments, the spraying is computationally controlled, so that the spray creates a tactile user-interface for the control of such devices as new musical instruments. These may be installed in public fountains to result in a fluid user interface to music by playing in the fountains. The invention may also be used in a setting like a karaoke bar, in which participants perform music by playing in a fountain while they sing. Small self contained embodiments of the invention may exist as pool toys, bath toys, or decorative fountains that can sit on desk tops, or the like.

FIELD OF THE INVENTION

The present invention pertains generally to a new kind of input/outputdevice that may be used to control a computer or a musical instrument,or that may itself be a device such as a musical instrument ormultimedia sculpture.

BACKGROUND OF THE INVENTION

Many traditional user-interfaces, human-computer interfaces, and thelike, are cold, mechanical/tegical and lack an expressive continuous“fluid” and immersive form of interaction.

Some user-interfaces, such as proximity-based, or antenna-based musicalinstruments like the Theremin, or “Doppler Danse” (Steve Mann “DopplerDanse”, Leonardo, Vol. 25, Iss. 1, 1992), achieve the desirable morecontinuous and immersive form of interaction but lack tactile feedback.

Likewise, “air typing” keyboards suffer from similar problems, as domany of the vision-based systems such as David Rokeby's “Very NervousSystem” (a vision-based system that uses a camera as an input device tocontrol virtual musical instruments, by tracking people's body positionin space).

Playing these instruments is very difficult because they provide no“feel” of where individual notes are located.

SUMMARY OF THE INVENTION

The following briefly describes my new invention.

It is possible with this invention to provide a more “fluid” as well asa more continuous and “immersive” multimedia input device with inputelements that a user can feel.

It is possible with this invention to provide a Theremin-like musicalinstrument or other input device but one in which the user has somefeel, that is provided by a fluid that is instrumental in theinteraction, or in which a tactualization is formed by a descrete set ofholes for notes.

It is possible with the invention to select from a discrete orcontinuous alphabet of symbols, using a body of air or water as an inputdevice, or using a discrete set of holes as an alternative form oftactualizer.

It is possible in embodiments that use a fruit, that this fluid can alsobe part of a closed-loop interaction.

It is possible that the fluid can be optically and visually engaging, aswell as tactile.

The following provides an informal review/summary of my new invention.

The invention can be incorporated into pool toys, bath toys, smalldecorative desktop/tabletop fountains, hot tubs, larger publicfountains, municipal swimming baths, the towers (platforms, such as a5-meter platform or a 10-meter platform) at swimming baths, as well asin small portable devices that can be connected to a garden hose, or toa small pump to draw fluid from an ocean, lake, hot tub, bath tub, orthe like.

One aspect of the invention allows a bather to press down on a spray jetof fluid, to play different musical notes, the notes depending on amanner in which the fluid flow is restricted. In addition to music,other functions such as a combination input/output (keyboard/display) inwater are possible.

One aspect of the invention creates a flat sheet of water that functionsas a “splash page” to display a web page, projected onto the flat sheetof water, such that a bather can touch part of the sheet of water toselect something from the web page.

Another aspect of the invention uses a pool as the splash page, withscoffing multimedia matter projected onto the pool, or the bottom of thepool, so that a bather can enter the pool (possibly with the entirebody, as, for example, from a 5-meter or 10-meter platform) to selectsomething from the pool.

The splash page can also be made from a two dimensional array of jetswith means for sensing restriction of individual jets in the array.

The apparatus of the invention allows the user to convey information ina very poetic, expressive, continuous, fluid way, and also forinformation to be presented to the user in a natural manner.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will now be described in more detail, by way of exampleswhich in no way are meant to limit the scope of the invention, but,rather, these examples will serve to illustrate the invention withreference to the accompanying drawings, in which:

FIG. 1A illustrates the principle and components of the invention whereseparate restrictometers are used.

FIG. 1B illustrates the principle and components of the invention wherethe restrictometers are the fluid supplies, with a separate fluid supplyfor each fluid jet.

FIG. 1C illustrates an embodiment of the invention with a separatehousing for each of a plurality of modules that each have a fluidsupply, fluid jet, and restrictometer, each combined with eitherwireless communication or sound producing device.

FIG. 1D illustrates the fluid diversion principle of the invention inwhich a fluid is diverted to cause sound production when a fluid jet isblocked.

FIG. 1E illustrates a purely acoustic version of the musical instrumentthat uses water as a user interface medium.

FIG. 1F illustrates the principle and components of a single-jetembodiment of the invention.

FIG. 1G illustrates an arrangement of jets suitable as an input devicefor a wearable computer, in which jets of compressed air form thetactile feedback mechanism, whereas the restrictometers measure opticalrestriction and operate entirely separate from the fluid flow.

FIG. 1H illustrates an arrangement of jets suitable as an input to agame that teaches children to sing at a constant tempo.

FIG. 2 illustrates how the single-jet embodiment of the invention may beused to convey a very expressive form of freely flowing, continuousinput data with fluidity.

FIG. 3 illustrates a multi-jet embodiment of the invention.

FIG. 4 illustrates the vacuum exclusion principle of multi-jetembodiments that makes flow diversion selectivity possible.

FIG. 5 illustrates an embodiment built inside a pipe such as a torusswim ring, inner tube, or other fully enclosed housing.

FIG. 6 illustrates a diversion of fluid for expression of subtle inputsthrough partial parallel streaming media.

FIG. 7 illustrates the principle of multi-jet fingering.

FIG. 8 illustrates a very simple way in which simple low cost sensorsand wiring can be made immune to the effects of water conductivity, aswell as a simple embodiment of the invention for use by inexperiencedusers.

FIG. 9 illustrates a platform embodiment of the invention that is afully and totally immersive multimedia environment.

FIG. 10 illustrates the timing diagram for an embodiment of theinvention that uses two jets to display, as well as to alter a one bitstate setting, or to interact (e.g. to have a watertight acrosscyberspace, to push water through the internet and out the other side,etc.).

FIG. 11 illustrates a splash screen waterjet impression pad.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

While the invention shall now be described with reference to thepreferred embodiments shown in the drawings, it should be understoodthat the intention is not to limit the invention only to the particularembodiments shown but rather to cover all alterations, modifications andequivalent arrangements possible within the scope of appended claims.

In all aspects of the present invention, references to “camera” mean anydevice or collection of devices capable of simultaneously determining aquantity of light arriving from a plurality of directions and or at aplurality of locations, or determining some other attribute of lightarriving from a plurality of directions and or at a plurality oflocations.

References to “processor”, or “computer” shall include sequentialinstruction, parallel instruction, and special purpose architecturessuch as digital signal processing hardware, Field Programmable GateArrays (FPGAs), programmable logic devices, as well as analog signalprocessing devices.

When it is said that object “A” is “borne” by object “B”, this shallinclude the possibilities that A is attached to B, that A is part of B,that A is built into B, or that A is B.

FIG. 1A illustrates an acoustic air-based embodiment of the invention.

One or more fluid chests 30FC supply fluid (such as air, or water) toone or more fluid jets 31. Fluid jets 31 may be nozzles, spray jets,water jets, air jets, etc., that can be interacted with by a user of theapparatus of the invention.

Each jet has, associated with it, a restrictometer 31R, that measuresthe degree to which the flow of the jet is being restricted. In someembodiments of the invention, the restrictometer provides a continuousmeasure of restriction, whereas in other embodiments the measurement isdiscrete (digital). When the measurement is discrete (digital) therestrictometer 31R senses which of a discrete set of restrictometricstates the the flow of jet 31 is in. For example, the restrictometer maysense that jet 31 is in one of four states: no flow, low flow, mediumflow, or high flow. In other embodiments, the restrictomter 31R simplysenses whether or not the jet is blocked, and thus has only two statesof sensory capability: on or off. In this situation, a flow switch is asatisfactory restrictometer 31 R. Alternatively, a pressure switch onthe side-discharge of a “T” fitting as commonly used in plumbingsystems, with the fluid going through the straight path of the “T”fitting, will make a satisfactory restrictometer.

The term “restrictometer” appears in the published scientificliterature. See for example, “Image processing considerations for simplereal-time fluid-based user interfaces”, Steve Mann, in Proceedings ofthe IEEE International Conference on Image Processing (ICIP), papernumber 1442, Lausanne, Switzerland, Sep. 11-14, 2005.

See also, “flUId streams: fountains that are keyboards with nozzle sprayas keys that give rich tactile feedback and are more expressive and morefun than plastic keys”, International Multimedia Conference archiveProceedings of the 13th annual ACM international conference onMultimedia, Hilton, Singapore Pages 181-190, 2005. ISBN:1-59593-044-2Author: Steve Mann; Sponsors: ACM: Association for Computing Machinery,SIGGRAPH: ACM Special Interest Group on Computer Graphics andInteractive Techniques; SIGMULTIMEDIA: ACM Special Interest Group onMulti-media Publisher, ACM Press New York, N.Y., USA. Alternatively, apressure sensor may be used on the side discharge of the “T” fitting. Anumber of flow meters are also suitable, such as a pinwheel flow meter,or even just a pump used in reverse as a generator to generateelectricity when fluid is forced through it, thus measuring how muchfluid is going through it.

If the sensing is binary (i.e. sensing only on and off states) it ispreferable that it have some hysterisis, which can be achieved byquantizing a continuous sensor appropriately, or by using a snap switch(microswitch) on a bellows, diaphragm, membrane, flow lever, arm, or thelike. Many magnetic reed switches also have hysterisis and aresubmersible. In the case of a magnetic reed switch, a simple diaphragm,to use pressure to move a magnet toward or away from the reed switch,will result in a suitable pressure switch. Alternatively, a flow switchcan be implemented by a small paddle or flapper that swings a magnetnear a magnetic reed switch to detect flow in the side-branch of the “T”fitting (i.e. to sense restriction of the main-branch).

Other sensory combinations for restrictometer 31R are also possible..For example, a diaphragm with a small mirror or other opticalarrangement to measure flexing of the diaphragm can be used with aphotocell and light source. Two or four photocells in a bridge can beused to convert from flow to resistance value, thus measuringrestriction continuously.

A piece of surgical tubing can also be used to measure restrictionbecause it will flex or bend when there are flow or pressure changes.

Finally, an optical restrictometer can be used, based on the opticalproperties of the fluid, especially with water, where the opticalproperties of the water cause it to act like a cylindrical lens. Therestriction can thus be sensed by cameras, photocells, light detectors,or the like.

When restrictometer 31R is continuous rather than discrete, theinstrument can be “velocity sensitive” like a piano, in which hittingthe jets harder results in a louder sound.

However, it is preferable that restriction continually affect theamplitude of the sounded note, rather than having the instrument bevelocity sensitive. In this way, the instrument works more like atracker organ keyboard than like a piano keyboard. Rather than breakingthe note down into initial setup by velocity, with possible furthermodification by aftertouch, it is preferable to have “duringtouch”, i.e.a touch that starts, and continues, to be consistent. This consistencyis provided by continually updating the note volume as a function ofrestriction. Ideally, therefore, all notes (or at least those over acertain restriction threshold) are always sounding, and the volume ofeach one is simply modulated with degree of restriction.

The restrictometer 31R can also be an expressive restrictometer thatsenses the way in which the fluid is blocked, such as for example todistinguish between. a hand that blocks it straight across and at anangle. A sonar, inside the fluid chest 30FC can, for example, “see” areturn from the hand 130 of a user of the device. The restrictometer canthus sense the height of the water jet emerging from jet 31, as well asthe manner in which it is blocked. Thus the restrictometer may, forexample, be able to tell the difference between a 6 inch (152 mm) jetthat is blocked straight at 3 inches (76 mm) and one that is blockedcrooked at the same height of 3 inches (76 mm).

These restrictometric nuances can be passed along to processor 140 tosynthesize a rich sound of a wonderfully complex musical instrument thatresponds to not just how far down a particular jet is pressed, but alsoto which way the jet is pressed. Thus the apparatus of the invention canwork like a tracker organ (an organ that responds to how far down keysare depressed) with further expression effects such as pitch bend byblocking a jet at an angle in the direction of desired bend. Thusspraying fluid to the left (by selecting the angle of the hand 130,tilting the hand) may, for example, cause a downward bend in pitch.Spraying to the right can raise the pitch. Spraying up and down (i.e.toward or away from the user) can cause other effects such as continuouschange in timbre.

When one jet is blocked, fluid 32 may emerge more quickly from otherjets. Processor 140 can account for this change, and solve a plumbingnetwork, using well known network solving algorithms, to make a moreaccurate inference of flow changes.

Alternatively, a separate fluid supply 30FS may be used for each jet, sothat there is not a sharing of supply, so that fluid chest 30FC iseliminated. For example, there may be a pump for each jet 31.

In the diagrams, the jets are shown as single jets, but, in order to putexpression into the music, the jets may be segmented. For example, a12-jet instrument may typically include 24 or 48 restrictometers (two orfour per jet), where the jet is segmented into halves or quarters, sothat blocking the left side of a jet can be read differently thanblocking the right side of the jet, etc.. Thus, for example, a musiciancan “bend” notes by blocking the left or right half of a jet. Blockingfrom top to bottom can change the timbre. Typically, blocking the bottomof a jet causes a deeper, more muted sound, or a more pure flute-likesound, whereas blocking the top of a jet causes a brighter more brassysound, or a more bombarde-like sound, richer in harmonics. By moving thefinger around the hole in different ways, a very richly expressive formof music can result. The resulting ability to sense a hydrodynamicflowfield can be used in different ways. Thus the sensory capabilitiesof each jet are multidimensional, with volume (amplitude) being on the“Z” axis (greater or lesser restriction along the central axis ofsymmetry of the round jet, for example), slight pitch bending (betweennotes) being affected by moving the finger a little bit along the “X”axis (side-to-side), and timbre being affected by moving the fingeralong the “Y” axis (up and down).

FIG. 1B illustrates an embodiment of the invention that does not use afluid chest. Instead there is a fluid supply 30FS for each jet 31. Senselines 30SL from each fluid supply 30FS pass along the information toprocessor 140 to indicate the degree of restriction on each of the jets31.

These sense lines are shown as dotted lines, because they are often notnecessary. Instead, a power supply 30PS that supplies the fluid supplies30FS (e.g. a power supply that supplies a separate miniature pump foreach jet) is an intelligent power supply that monitors power consumptionon a per-pump basis. In this way, the pumps are each their ownrestrictometer, such that when a particular pump is blocked, theelectrical consumption or other characteristics of the pump aremonitored, and this information is used to sense the restriction offlow.

As an example, a small 12 volt submersible pump may draw 6 amps currentwhen not blocked, but only 4 amps current when blocked the one thirdreduction in current can be sensed to trigger a note of a pitch thatcorresponds to a particular note on a musical scale in keeping with theposition of the jet in a row of jets. The note can be sustained for aslong as the jet is blocked. The volume of the note can be adjusted, forexample, to full volume at 4 amps, to half volume at 5 amps, and to zerovolume at 6 amps current draw by the pump associated with thatparticular note. If this affine relationship of current versus volume isnot desired, a LookUp Table (LUT) can be used to shape the note volumeas a function of flow restriction.

Power consumption of other devices can be similarly used. For example,power consumption of a steam boiler, ultrasonic atomizer, or the like,can be monitored to estimate restriction.

Alternatively, if the water is being heated, separate on-demand heatersfor each jet can be monitored to estimate flow restriction. Heating ofthe jets is sometimes desired to make the instrument more comfortable toplay.

In multi-pump embodiments of the invention, there can also be more thanone pump per jet, in order to sense a hydrodynamic flowfield. Forexample, a 12-jet water-based instrument may have 48 pumps, with fourpumps per jet, each group of four being arranged to spray into aquad-segmented jet.

FIG. 1C illustrates an embodiment of the invention that uses a separatehousing 98 for each note. The housings might, for example, be flowerpots, or similar pots as are commonly used for small decorative tabletopor desktop fountains. Each housing has a fluid supply 30FS which might,for example, be a pump, to spray fluid 32 out jet 31. The user touches,for example, by way of hand 130, each jet in succession to type or playmusic or for other forms of interaction.

The units housed in housings 98 may each contain a sound making device,of a specific pitch. For example, with 8 housings 98, a musical scale,such as A, B, C, D, E, F, G, a (natural minor) or C, D, E, F, G, A, b, c(major) may exist in the choice and design of soundmaking devices ineach of the eight stand-alone units.

These can be manufactured as a set, or sold individually, so that acustomer could buy the notes that he or she would like to have, andarrange these in a desired musical scale along a desktop. Each unit mayhave it's own batteries, if desired.

The units may have wireless communication that could be used to set thenote's pitch for each unit.

In other forms of wireless communication, the units may interact ininteresting ways. For example, two units may interact in a playful way,rather than as a musical instrument. When the user pushes down the jetone one, the other's jet turns on, and vice versa. When separated overdistance, e.g. on two separate desktops of colleagues, co-workers, orspouses, the result is a playful way of interacting. With wirelessrepeaters, Internet connection, or the like, the interaction can embodya kind of waterfight across cyberspace, where there is a simple metaphorof “pushing water through cyberspace”.

With this embodiment, or various other embodiments of the invention,radio buttons can be implemented in which all but one jet 31 initiallysprays water, and pressing another jet causes that other jet to staydown. This feature could, for example, select from among various radiostations. For example, if a person had eight favorite radio stations,there could be eight jets but with only seven of them running. The onethat's not running corresponds to the radio station playing. Pressinganother jet down changes to that other radio station.

With wireless control of lighting and other equipment, the invention maythus be used to switch various lights on and off. For example, two jetsmay be used, so-that pressing down on a first jet causes the first jetto turn off and a second jet to turn on, as well as the lights in theroom to turn on. Pressing down on the second jet causes the second jetto turn off and the first jet to turn on, as well as causing the lightsto turn off.

FIG. 1D illustrates an acoustic air-based embodiment of the invention,showing just one note. A fluid chest 30FC delivers compressed air to anumber of fluid jets 31. This can be accomplished by tapping into fluidchest 30FC with a reducing “T” fitting 49 for each tap point. Forexample, if there are to be 25 notes (for a 2-octave chromatic range),there would be 61 reducing “T” fittings 25, spaced along fluid chest30FC.

The outlets from each of these reducing “T” fittings 49, are eachconnected to a regular (non-reducing) “T” fitting 40. Regular “T”fittings 40 are of a size compatible with the side outlet of reducing“T” fittings 49. For example, fluid chest 30FC might be a one inch(approx. 25 mm) copper pipe. Reducing “T” fittings 49 might be one byone quarter by one inch (approx. 25×6×25 mm) fittings, and regular(non-reducing) “T” fitting 40 might be a quarter by quarter by quarterinch (6×6×6 mm) “T” fitting.

When hand 130 descends to partially restrict flow out of one of 25 jets,such as jet 31, thus reducing the amount of air that comes out of thatjet, a greater portion of air is forced out side discharge 41, thanwould be forced out when hand 130 is not present.

As jet 31 is restricted to a greater degree, more fluid (air in thiscase) flows out through side discharge 41, into a flow-based soundproducing device 99. A satisactory sound producing device is an organpipe, as commonly used in a pipe organ, such as the pipe organs commonlyfound in churches, convention centers, skating rinks, and the like. Thesound-producing devices may be whistles, flutes, or other sound-makersthat make sound when air is fed to them.

There may be various embodiments of the invention having various numbersof notes. In a 25 note (2-octave chromatic) version of the invention,there would be 25 sound producing devices 99, each tuned to theappropriate note.

FIG. 1E illustrates an acoustic water-based embodiment of the invention,having eight notes on a diatonic natural minor (aeolean mode) scale: A,B, C, . . . a. Only four of the eight notes are shown: the first three(A, B, and C), and the last note (a). In this usage, fluid chest 30FCcarries water to water jets, from which emerge water 31F except jet 31,when and where flow is blocked by hand 130. Alternatively, flow may bepartially restricted by hand 130, to vary the amount of water squirtingout through side discharge 41.

Each side discharge 41 directs water at the inlet of a pump 30P. Theside discharge 41 is not sealed directly to the inlet of pump 30P, but,rather, there is an air gap, 30AG between the side discharge 41 and theinlet of a pump 30P. Pump 30P is a pump that can run wet or dry. Whenwater is squirted into its inlet, it pumps the water into a steam boiler30B. For each note, there is a miniature steam boiler 30B, to supplysteam to a sound making device such as device 99A. In this example thereare 8 boilers, 4 of which are shown in the drawing. A suitable device99A would be a pipe from a steam calliope, steam organ, steam whistle,or the like. Each of the 8 boilers, such as boiler 30B, are heated byflame 30F from heat source 30H.

The purpose of pump 30P is to overcome the pressure P_(b) in the boiler.Thus pump outlet pressure P_(o) should be greater than pressure P_(b) inthe boiler. However, if pump inlet pressure P_(i) can be made higherthan pressure P_(b) in the boiler, then pump 30P and air gap 30AG arenot necessary in this case, water is squirted directly into the boiler.A typical scenario might be to use calliope pipes that take very lowpressure, such as 2 inches (approx. 51 mm) of water column, so that thewater squirted out of side discharge 41 has sufficient pressure tosustain a note without the need for pump 30P and air gap 30AG.

Otherwise if the boiler pressure is too high, notes cannot be sustained.A one-way valve can be used, instead of the pump and air gap, if notesof short duration are acceptable.

When a jet such as jet 31 is blocked or partially blocked, water issquirted into a pump such as pump 30P to feed boiler 30B to sound device99A. The water is convereted into steam in the process, resulting in anice visual effect, as well as the sound.

If desired, a hybrid acoustic/electronic instrument can be made in whichsteam is produced, while at the same time triggering a note. This may bedone with electronic ultrasonic atomizers in place of boiler 30B, flame30F, and heat source 30H. Pump 30P can also be eliminated, so that thewater is squirted directly onto the atomizer.

Steam atomizers can also be used as sensors. For example, notes can betriggered on the presence of steam, by way of optical, conductivity, orother sensors.

Some atomizers come on automatically when squirted with water, in whichcase the electrical load change can trigger notes. For example theatomizers can be plugged into a special intelligent power bar thatsenses electrical draw from each outlet, and activates notes when thereis electrical draw. Thus notes get sounded electronically when steam isgenerated or when steam is called for.

Since many atomizers are ultrasonic, the ultrasonic waves can also beused to directly sense the position of hand 130, thus eliminating a needfor jet 31 and side discharge 41. Instead, the hand position in thewater is determined by the ultrasonic wave, using the ultrasonic disk inthe atomizer as both a sender and receiver of sonar. Sonar devices mayalso be installed in the jets 31 to make a hybrid electronic/acousticinstrument, and various sensory combinations.

Alternatively, RADAR (Radio Direction and Ranging) or LiDAR (LightDirection and Ranging) or some other form of energy may be transmittedand its reflection measured, in order to determine hand position. In avery simple embodiment of the invention, a photocell or light sensor maybe installed in each of a plurality of holes of a pipe, so that thesystem measures restriction of the flow of light. Such a restrictometermeasures how much light is restricted, independent of the flow of fluid.Thus the presence of the fluid-in-motion (e.g. air or water jets) can bethere just for tacticle feedback, while the actual sensing is done withphotocells. Satisfactory photocells are photoresistors such as CadminumSulphide (CdS) cells. The resistance of the cell decreases as the lightto the cell increases. Therefore, a sensor senses the increase inresistance as the cell is blocked, and increases the volume of (orabruptly turns on) the note corresponding to the hole corresponding to aparticular photocell. Some embodiments of the invention can use ambientlight and the attenuation thereof, in which case a dummy photocell in awheatstone bridge helps to mitigate the adverse effects of changes inambient light levels.

Alternatively, an L.E.D. (Light Emitting Diode) and a photoreceptor(such as a photodiode or phototransistor) may be used, to measurereflected energy from a user's hands or fingers, or the like, placedover each of the holes. In this case, the fluid may also be used justfor tactile feedback, independent (if desired) of the restrictometry,where said restrictometry is a measure of the degree of activerestriction (restriction of a source of light from within). Typically inthis autorestrictometric (self-restrictometric) embodiment the light isinfrared. Typically, in this embodiment, the outgoing light ismodulated, so that some kind of encoding (whether by a simple lock-inamplifier, or more sophisticated coding) allows the system to be somewhat immune to changes in ambient light. Therefore, in much the same waythat automatic flush urinals and automatic hand wash faucets ignoreambient light, the musical instrument of the invention can also ignoreor be less affected by ambient light.

FIG. 1F is a diagram depicting a fluid jet 110 that sprays water in theair from a ground nozzle 111, flush with ground level and at groundpotential, electrically connected by ground 112. A satisfactory groundnozzle is a laminar flow nozzle, such as the nozzles made by WETDesigns, since the laminar flow has desirable optical properties, forthe recognition of the height of jet 110 by a computer vision system. Asystem for dispensing and animating the water in packets, such as by wayof nozzles often used for decorative fountains (like the fountains infront of the Brooklyn Museum that provide dancing jets of water) isdesirable in some embodiments of the invention, so that the water can befinely controlled as part of a user-feedback loop, embodied in processor140, through flow controller 113. Water dispensers known by the tradename “Jumping Jacks” may also be used.

Splash, spray, and jets of water tend to look very beautifully intensewhen backlit, such as when one looks at fountains when the sunlight isbehind them, or when people splash into a pool with the sun behind thedroplets of water. This is because the droplets behave like a lens,though poorly, in terms of optical quality, but good enough toconcentrate the sun's rays over a range of angles where at least some ofthe water caustics (loci of points of what would be infinite brightnessin simple theory) are visible slightly off-axis.

A bather blocks a portion of the jet 110, for example, with their hand130, to prevent the jet from going beyond a certain height. Generallywhen a bather inserts his or her hand 130 into the jet, the water willhit the hand and crash back down mostly, with various droplets sprayedin the area.

A light source 120 serves to backlight the jet 110 with respect to anoptical sensor 150, in order to measure the height of water column ofjet 110 by way of processor 140. A baffle 170, either as part of lightsource 120, sensor 150, or a combination of both, or as separateelements, keeps light from shining from light source 120 into sensor 150even though both are opposite jet 110, except, of course when and wherejet 110 is spraying. This arrangement causes there to be almost completedarkness where there is no water, but almost complete whiteness wherethere is water. Thus the user's hand 130 will appear in silhouette, as ablack outline, along with the user's body, and other objects, but thewater jet, and all the droplets of water, will be bright white.

A satisfactory optical sensor 150 is an ordinary video camera, whereinprocessor 140 may be equipped with a frame grabber so that it cananalyze the image of the backlit jet 110 and determine the highestbright spot, which corresponds approximately to the highest that the jetwas allowed to go by hand 130.

A good kind of light source 120 is an aircraft landing light, or a PAR36 or PAR 38 pinspot light, as are commonly used at rock concerts, andin theatrical lighting, to create dramatic strongly collimated lightthat emulates natural sunlight. In addition to providing an ideal lightsource for the computer vision system of sensor 150 and processor 140,such light creates a very pleasant “summertime” atmosphere that makesthe invention comfortable to use in cooler weather, since the stronglycollimated light has both an actual (concentration of heat rays) as wellas psychological warming effect. Thus playing in the water jet is warm,and the living is easy, in the sense that a bather can feel nice andwarm while playing.

In case there is actual sunlight that might illuminate backgroundobjects such as glare off windows in the background that could falselyactivate the optical sensor 150, a lock-in amplifier may be used in thesystem to improve signal to noise ratio, or some other form of lightmodulation may be used, with controller 121. A satisfactory controller121 is a bidirectional triode thyristor based control, such as by way ofa triac-based light dimmer, although IGBT-based light controllers thatcan generate more arbitrary waveforms are more desirable. An insulatedgate bipolar transistor (IGBT) is preferable as it combines the highcurrent density characteristic of a bipolar junction transistor with thefast response and better output characteristic typical of an insulatedgate field effect transistor (e.g. MOSFET). The ability to generatearbitrary waveforms is useful for signaling and modulation schemes.

In general, some form of light modulation, lock-in 160, puts a messagesignal onto the light, so that fluctuations in the light bear themessage, whether it be a sinusoidal variation of light output that rideson a DC (Direct Current or other average) offset, or some otherencoding. Ideally an adaptive encoding is used, as necessary, tomodulate the light for good signal to noise ratio.

In the interest of modulation, a tungsten aircraft landing light, or thelike, may be less desirable than a light source 122 that is based onLight Emitting Diodes (LEDs), especially since LEDs can be arranged in alinear array parallel to the jet 110, and also can be toed in to allpoint at optical sensor 150. This results in more efficient use of lightas well as a better exploitation of the lenslike properties of the waterjet 110. In so far as the jet 110 behaves optically similar to a glassrod, its properties may be best exploited with the arrangement ofsources 122, as high brightness LEDs that have very narrow field ofillumination (i.e. that concentrate most of their optical energy along anarrow axis) arranged to point as depicted by the arrows. Thus thelights furthest to the ceiling point downward, whereas the lights nearthe floor point up. This way, each section of jet 110 is perfectlybacklit.

In typical usage, a bather may interact by pressing down a water jet110, or by swinging the hand at the jet 110, or by taking a swipe at it,or even pulling a piece out of the middle of the jet 110. This action issensed, and results in some outcome, typically given to the user by wayof some feedback. The bather's interaction will typically select from adiscrete alphabet of symbols, much like a “QWERTY”. keyboard on acomputer, or an “ABCDEFGABCDEFGABC . . . ” keyboard of a piano.Additionally, this alphabet may be a multidimensional alphabet in thesense that each symbol may have meta information in it. On a computerkeyboard, when we type the letter “A” there is no emotion carried withhow hard and angry we hit the “A” or when. A piano carries more metainformation with the key. Accordingly, the invention allows even moremeta information than with the piano keyboard. In addition to velocity,force, displacement, and timing profile, the multidimensional alphabetselector of processor 140 can measure subtle nuances of the way in whichthe letter “A” is plucked from the column of water jet 110.

Once a symbol is chosen from the plurality of possible symbols, thissymbol may then take action in feedback to the very input device thatthe symbol was plucked from. Unlike a computer keyboard, or even a pianokeyboard, there is a programmable closed loop feedback system thatmodulates the very input medium.

Consider, for example, a simple task of adjusting water flow using thenew input device. For example, the water jet 110 can simulate quantizedstates of height, and remember height, where the user can adjust theheight of the jet, by hand. If the user wants the jet to run low, theuser simply pushes the water jet down, and it stays down when the userwalks away. If the user wants the jet to come back up, he or she walksover to it again, and grabs the jet 110 and pulls it up. In thisapplication, the jet sprays up until it encounters the user's hand, andthen stops. The system can detect the user's hand in a variety of ways,either directly by computer vision, or more preferably, by a betterclosed-loop process in which:

1. water jet 110 height is initialized to zero by setting controloutputs from processor 140 to control inputs to a combination of nozzle111 and controller 113 such that the flow is zero or sufficiently low;

2. jet 110 height is incremented by applying ever increasing amounts offlow, by appropriate adjusting of outputs from processor 140 to controlinputs to a combination of nozzle 111 and controller 113;

3. the transfer function between a jet control input signal 115(consisting of control outputs from processor 140 to control inputs to acombination of nozzle 111 and controller 113) and the height of the jetis adaptively modeled (having been determined previously but withadaptation to varying wind, varying water characteristics, and thelike);

4. a change in the transfer function characteristics is continuouslychecked for;

5. if increments to jet control input signal 115 do not result insufficient increase to the height of jet 110, then it is assumed thatthe jet is blocked, such as by hand 130;

6. the height at which the jet is blocked to is continuously monitored,while continuously checking for unblocking of the jet;

7. when the jet is unblocked, it is controlled actively to remain at theheight at which it was last unblocked. This controlling is done in aclosed loop fashion, by maintaining the height with jet control inputsignal 115 being adjusted to keep the jet at that height despite driftdue to changes in water pressure, wind, and the like.

In a preferred embodiment, whenever no bather is detected (i.e. noblockage by hand 130 is detected) the jet 110 rises and falls in asinusoidally periodic fashion in order appear playful and enticing. Inparticular, many fountains have rising and falling jets which are foundto be quite pleasing. For example, the architectural and artistic focalpoint of Canada's cultural and civic epicenter (known as “Times SquareNorth”) in Toronto's Dundas Square is Dan Euser's sculpture whichconsists of 600 ground nozzles (arranged in 20 grilles with 30 nozzleseach) that spray water up in a rising and falling way to mimic the waveson a beach, or the pounding surf of the ocean. (Inhttp://wearcam.org/dundas-square/ there is an explanation of DundasSquare's existing waterplay nozzle jet sequencer.)

This provides a soothing sound that masks traffic, while inviting peopleto play in the water.

Thus the present invention can be used in similar kinds of places, tocreate the same kind of rising and falling surf, but while also beingresponsive to input from users. The present invention allows people tosculpt the water, and have full playing in the fountains while shapingthe water flow through play.

In preferred embodiments, the rise and fall continues but with reducedamplitude, when jet 110 is blocked, and the continued oscillation ofheight of jet 110 is in the vicinity of the blockage, so that the riseand fall can be used to advantage as a way to more accurately measurethe response effect of the blockage.

In an alternative simpler embodiment, the jet 110 can simply be poweredmore than where blocked, as previously known by the transfer functionbetween signal 115 and height. Thus it can simply then be known thatblockage has occurred when the height is less than it should be for agiven signal 115.

In either the preferred or simplified embodiment, the signal 115 ispreferably dynamically varied against the blockage of hand 130, toprovide a time-varying tactile feedback signal to the user. This can beused to send back a “buzz” that the user feels upon the hand, much likethe vibration of a silent pocket pager or cellular telephone vibrator.

This vibration can vary in pitch, amplitude, waveshape, and chirpiness,etc., as a way of providing user feedback as a variety of user feltsymbols, either from a discrete “dictionary” or as a more continuouslyfelt form of water expression.

Additionally, since the ground is wet, and since water that has beentreated with salts, chlorine, bromine, or the like, is very conductive,a return path through the user may also be detected along with otheradditional optical properties, such as a change in the color of the jet110. Especially if the jet 110 is laminar, it behaves like a fiber opticinformation conduit, and the flesh color of the hand is visible insidethe jet, as an additional measurement signal 116. Thus processor 140 hasvarious ways of detecting and measuring the presence, position,orientation, and the like, of hand 130.

Additionally processor 140 measures the way in which water is swept awayby hand 130, so, for example, smashing through the jet 110 to push waterto the north can result in different action than pushing east, west, orsouth. Pushing the water up and to the north can take different actionthan pushing it down toward the ground and to the north. Thus thedirection of entry of the hand, as well as the direction that the wateractually splashes, can affect the Wet User Interface (WUI), Fluid UserInterface (FUI), more specifically, typically a Liquid User Interface,LUI. In large installations like public fountains that are alsoimportant architectural landmarks, it may be desirable to have multiplesuch jets 110, each differently colored by lights inside nozzles 111.Thus a whole array of beautiful dancing fountains can be set forth thatcan be choreographed by automation that is adjustable by people playingin the fountains. Each jet can also be a separate symbol area forselecting from a discrete alphabet of symbols out of each jet, or out ofthe ensemble, or any combination thereof.

In this case, light source 120 may be a source that tracks and follows abather, to backlight whichever spray jet 110 the bather decides toactivate next. Followspot technology in which a spotlight follows astage performer as he or she moves around, is well known in the art.Thus an automated followspot may be used as both a vision aid for theoptical sensor 150, as well as to keep the bather warm, and illuminated,as might be desirable in an interactive art installation. Alternativelyif it is desired for the vision light to be invisible, an infraredaircraft landing light, or the like, can be used. A satisfactory suchlight source 120 is a dichroic PAR 56 (Parabolic Aluminized Reflectorsize 56) infrared heat lamp or similar light as often used in securityapplications. This will still serve to keep the bather warm, and toprovide illumination for the vision system including optical sensorsensor 150. Thus the bather can freely move around in a large waterplayarea and interact with various jets.

For example, all six hundred of the jets in Dan Euser's masterpiece atDundas Square could, in principle, be made to rise and fall in responseto one person inserting their finger into one of the jets. Thus simplytouching one small spray of water would result in a chorus of thunderfrom the other 599 nozzles. Children and adults alike would thus take amoment from their walk through the Square to stop and touch the water,and create dynamic art. This touch to the water could also affectmomemtarily the billboards and giant pixelboard displays. Whilemomentarily interrupting the advertising for art's sake, lost revenuecould be made up for by the fact that more people would be looking up atthe pixelboards because they would be truly interactive extensions ofthe water spray as their input. For example, pressing down on the nozzlejets could cycle through various ads, making the nozzles function likebuttons on a TeleVision remote control. This would create a publicinteractive waterplay art installation in which the water jets becomeinput devices, much like the keyboards and pointing devices ofcomputers. An omnidirectional jet could also spray in various directionsuntil blocked, and thus direction could replace heigt, or could beanother parameter in addition to height, of jet 110. This can be used asa pointing device in place of a computer mouse or trackpoint, and canalso be more expressive by including the two dimensions of cursorposition in addition to other dimensions like the three dimensionalspace plus the fourth dimension of orientation, and more (includingmultidimensional hand position, orientation, etc., not to mention alsothe wonderful tactile feedback that the immersive nature of water sprayprovides.

Fountains could also be internetworked, i.e. fountains that are too bigto safely play in (such as the fountains in front of the Bellagio Hotelin Las Vegas) could be controlled by a smaller waterplay fountain.

In this way a small child could choreograph the Bellagio fountains byplaying in a smaller fountain.

Such a large and expansive show presented from an individual couldfunction much like a karaoke machine, in the manner in which anindividual person of ordinary talent could “give” an excellent anddramatic show or performance. To the extent that karaoke is defined as a“method for the intoxicated to embarrass themselves” (Wikipedia.orgonline encyclopedia) playing in the fountains can further the fear ofsinging in public with the added fear of being seen in a bathing suit(or underwear) in public. In this sense, interactive waterplayperformance spaces could be installed in “watering holes” and otherdrinking establishments like restaurants, lounges, hotels, and bars.

FIG. 1G illustrates an arrangement of 12 jets, suitable as an inputdevice for a wearable computer. The jets, beginning from jet 1AJ in theupper right corner, are arranged in three columns of four jets in eachcolumn. Inside this air jet hole there is a photo detector, 1AD, and aphoto light source 1AL. Light source 1AL and detector 1AD, together withother circuits and processing (said circuits and processing well knownin the art of automatic flush toilets, automatic faucets, etc.) compriserestrictometer 1A. A satisfactory restrictometer may be made from asingle 4-wire package that contains a phototransistor and a LightEmitting Diode (L.E.D.). Other restrictometers; shown as 1A, 1B, 1C, and1D, form the first column depicted at the right. The next column iscomprised of four more restrictometers 1E, 1F, 1G, and 1H. These eightrestrictometers are supplied to a wearable computer that synthesizes thenotes low-A; B, C, D, E, F, G, and high-a, in response to restriction oflight. The circuits are arranged so that sounding of the notes beginswhen a finger is within one half to one quarter of an inch (onecentimeter or so) of any of the restrictometers 1A to 1H. The eightrestrictometers 1A to 1H are connected and programmed to sound thecorresponding notes of the natural minor scale, from low-A to high-a, sothat simple melodies like Summertime, The Ants Go Marching, The Cat CameBack, America I Love you So, Napoletana Tarantella Dance, etc., can beplayed, by successively blocking the light leaving sources such as 1AL,so that the light is blocked and reflected back to detectors such as1AD.

FIG. 1G shows the front of the “keypad” facing the user, but in actualuse, a wriststrap is provided and the keypad faces away, with thefingers curved around, in the same way that a person would hold aTwiddler. In fact, the first eight notes A-H are the same letters of theTwiddler product that is manufactured by Handykey Corporation. The lastcolumn gives the notes high-b, high-c, high-d, and high-e. Each row isseparated from the previous or next row by an interval of a perfectfifth, so, for example, going across from restrictometer 1A, torestrictometer 1E, moves up a perfect fifth. Air holes for jets such asjet 1AJ, allow puffs of air that are dynamically controlled as tactilefeedback. In simpler embodiments, a steady stream of air will oftensuffice as the feedback mechanism. Thumb switches 1bSW and 1oSW reducethe output frequency by one semitone, and one octave, respectively.Thus, for example, to play a b-flat, a user restricts the flow (ofescaping light) from restrictometer 1B, while simultaneously holdingdown the thumb switch 1bSW. The other thumb switch 1oSW serves to extendthe range of the instrument, so that it covers almost three octaves.

Some simple chords can also be played by restricting multiple jets atthe same time. For example, simultaneous restriction of restrictometers1A, 1C, and 1E, results in an a-minor chord, whereas simultaneousrestriction of restrictometers 1C, 1E, and 1G results in a c-majorchord.

The general embodiment depicted in FIG. 1G can also work in the absenceof the tactility of the fluid. More generally, many embodiments of theinvention include some form of tactualizer, in conjunction withrestrictometers. In the absenoe of fluid, the tactualizer is the holesthemselves which can be felt, when used with the restrictometers, tocreate a flutelike experience for the user. FIG. 1H illustrates anarrangement of jets 1JET suitable as an input to a game that teacheschildren to sing at a constant tempo, by way of Liquid Crystal Displays1LCD, as the output device of a computer system with the jets as input.The game pad can be incorporated into splash pads, spraygrounds, publicpools, and the like. The jets are arranged in a square pattern, on arubberized surface, so that as a user stomps on top of each jet insuccession, a song, such as Gershwin's 1935 lullaby, Summertime”, isplayed. Two or more players can also stomp around the square, goingcounter-clockwise, as they play. For example, a player stomps on theword “SUMMER” with his or her left foot, and then the next jet with theright foot, and then the word “TIME” with the left foot, and so-on,eventually walking around to the word “AND” which is hit with the leftfoot, and later the word “EASY” also hit with the left foot. The lullabyplays through a speaker mounted in the center of the pad, and if theplayer stomps at the right tempo, points are awarded, and the musicplays louder and stronger. If the tempo is a little off, the musicadjusts to match the tempo of the user, but the music turns down involume to indicate the timing errors. Additionally, the Liquid CrystalDisplays LLCD can dynamically prompt the player through the song, andoffer help in response to errors in temp. The line of the first verse ofthe song then replaces “SUMMER” with “FISH” (the lyrics of the secondline), etc. Finally, after the first verse, the words “FISH”, “HIGH”,etc., on the Liquid Crystal Displays 1LCD, change to “THEN”, “SKY”, etc.

FIG. 2 is a diagram depicting a splash screen or splash page 200, thatconsists of imagery projected onto a sheet of water that is sprayed froma flat nozzle 210. Here a sheet of music is projected and thus presentedto the user, as streaming media, in addition to or instead of jets 110.As an addition to jets 110, splash page 200 may fall behind jets 110, asa wet (and thus immersive) projection surface that the user can read,look at, or choose to ignore (and even “bump” into, walk through, orstand in). The user can refer to the splash page 200 from time to timein order to help remember the words and notes of a song, such as the1935 lullaby from Porgy and Bess (Gershwin, “Summertime”). The notes inthe lullaby are bounded from C to C, since the user has selected the keyof C minor to match is or her vocal range. This selection has been madewith hand 130 to block the height of jet 110C to a height thatcorresponds with the highest C note, the first note of the song. Thusblocking the water spray forms a liquid user interface into thestreaming media.

In response to that selection, the processor 140 has caused the words ofthe song to display in a manner appropriate for a C minor key, so thatthe user can sing along with the song, displayed in a karaoke fashion.Additionally, the notes themselves are displayed in a similar way, sothat the user can play the notes while singing. The notes are playedusing the Liquid User Interface, LUI, formed by water and theinteraction with the water, i.e. a note is sounded based on when, where,and how forcefully the user touches the water. The nature of the water'spath once it is deflected, also affects the way the sound occurs. Notonly can the user “pitch bend” notes (such as near the end of thelullaby, on the word “don't” in “hush little baby, DO-N'T you cry” whichbends down a minor second) but the user can also affect the nature ofthe sound by hand position.

In the preferred embodiment, the hand position is sensed by thedirection the water sprays off the hand 130, such that the sense of handcontrol is very intuitive because it is then consistent with the overallphilosophy of the Liquid User Interface, LUI.

For example, if the user tips the hand 130 so it is angled up and to theright, the water jet 110C will splash against the hand and water willsplash off to the right. Thus, in addition to sensing how high the jetwent before it got blocked by hand 130, this direction of splash will besensed by optical sensor 150, with processor 140.

To play the song, the user pushes the jet down to get lower notes, andlets up on it to get higher notes. The jet 110Ab is shown pushed down toaffect a change in pitch downwards by a minor third from where it is in110C. This change also operates as a closed-loop feedback system, sothat pressing down on the jet 110C to jet 110Ab results in a feelinglike a fret or similar disturbance at B-flat, Bb, along the way.

More generally, the present invention includes various forms of tactilefeedback, so that, for example, pushing down on jet 110 results in atactile sensation that is, in this example, achieved as follows,operating in processor 140:

-   -   sense height position of hand 130 along the height of jet 110 by        way of height sensor 240 which may comprise optical sensor 150        in conjunction with processor 140, or which may be a separate        height sensor;    -   compare height with an indexed list of virtual water fret        positions;    -   when one of these water fret positions is reached, provide extra        stimulation of the hand 130 with the water, through stimulator        241, for as long as hand 130 is within a certain tolerance of        the height position;    -   repeat.        The above describes a rectangular tolerance window, but in        actual preferred embodiments, a Bartlett, Hanning, or Hamming        window is used so that the virtual fret has rounded edges rather        than square edges. Additionally, the water always provides some        sensation, but the extra “buzz” of riding right on top of a        fret, is created by modulating the water spray in a fine burst        of rapidly changing levels, so that the user can feel each note,        as if a softly quantized instrument like a guitar were being        played on real frets that do such quantization. This creates a        Theremin-like experience but without the lack of tactile        feedback. An alternative or an additional form of feed-back may        include small electrical impulses in the water, changes in water        color (as by lighting, etc.) changes in water temperature (as        by, for example, alternating hot and cold jets of various duty        cycles), Additionally, one water source may provide feedback for        actions done on a different water source, for example, splash        page 200 may be an output device for input from jet 110.

The sheet of music is projected onto the sheet of water which may alsobe a touch sheet, that functions like a touch screen, so that as theuser touches the sheet, the coordinates of the place where the sheet istouched are sensed. This can work in addition to jet 110, or it cancompletely replace jet 110. When not using jet 110, the splash page 200becomes the primary user interface.

There may also be a switching back and forth between the two modes ofuser interface, e.g. a novice user who wants the splash page 200 tostay, may interact with it, whereas by an appropriate gesture of pushingaway at it with both hands, it goes away. Splash page sensors arepresent to detect when it is pushed away, either by both hands or thewhole body of the user. Thus the splash page can be just anintroduction, or for instructions, that goes away when the user isfinished with it.

FIG. 3 is a diagram depicting a multi-jet wet-user-interface, Nine jets310 spray water upward, tilted slightly toward the middle. A ringmanifold 300, having a diameter, in the preferred embodiments, thatranges from 20 inches (approximately 51 cm) to 2 meters, has a FemaleGarden Hose Thread (GHT) connector 301 on a “T” fitting that supplies itwith water in both directions. Each nozzle jet 310 is supplied from bothdirections with water. The entire manifold 300 and jets 310 may besupplied by fresh water from a garden hose, with runoff going toirrigation, such as when playing in a garden, or it may be supplied bywater from a batter operated pump such as a bilge pump used in marineapplications. Capacities of bilge pumps are usually specified in gallonsper hour; preferred embodiments of the invention work well with bilgepumps in the capacity range of 500 GPH to 2000 GPH, with highercapacities being sometimes preferable for dramatic show, of the spray ofthe invention, but not usually necessary for good functioning. Inparticular, the preferred capacity is around 1000 GPH.

In a preferred embodiment of small size (e.g. 20 inches, or approx. 51cm diameter), the pipe size for the curved pipes of manifold 300 is 1/2inch plumbing which is equivalent to ⅝ inch refrigeration (plumbing isspecified as inside diameter but the refrigeration industry specifies byoutside diameter). This size is suitable for being worn over the rightshoulder, so that the high notes are to the right, and near the top, andthe low notes are to the left and near the bottom, in front of the bodyof the user.

Jets 310 may be made by cutting an appropriately curved pipe intosections and then rejoining them with reducing “T” fittings. Suitablereducing “T” fittings are ½ inch through to ¼ inch (⅝ inch through to ⅜inch in refrigeration sizing). A piece of size ⅜ plastic toilet or sinkhookup sleeve fits nicely into each opening in the reducing “T”, with agood friction fit. Thus a module 311 may be built around the plasticsleeve, and inserted into each hole as needed. In this way, an entiremodule can be quickly replaced in the field. Module 311 is a flowsensor, and may also perform the role of an output device, such as flowcontrol, or other stimulus to the user. At the very least, module 311should measure the amount of flow, and thus facilitate a continuousfluid user interface. In this particular embodiment, each jet isassociated with a different note. Each note may be thought of as asymbol selected from a discrete alphabet of symbols, and each jet may beconsidered therefore as a symbol area, or a region around a symbol area,in which the symbol is selected by having the user enter this area.Movement between symbol areas results in the generation of an orderedlist of symbols that are also annotated. The annotated ordered list usesannotation to record time of entry and exit to and from the area, andvarious attributes of how the entry and exit was made.

Each note sounds in amplitude that depends on how far down the jet forthat note is pressed. For example, if pressing down the “C” jet, the Cnote will sound and the sound will grow louder as the jet is pressedfurther down. To play a C-Major cord, the C, E, and G jets are all threeblocked together. To play a C note with a C-Major to accompany it, the Cjet may be blocked entirely, and the E and G only blocked slightly, orthe fingers may hover above the E and G, just lightly in the spray,whereas the finger may reach deeper down into the spray of the C jet.

It is preferable that the notes are activated by displacement ratherthan velocity, but if velocity is desired, the height value may bedifferentiated by processor 140. Since it is easier to take reliablederivatives than integrate reliably (due to the presence of baselinedrift), the absolute height measurement of each jet is preferable to thevelocity information.

The default setting for the instrument is also in displacement, andbehaves much like a church organ, which is also easier to sing to thanthe more percussive and more ephemeral sound of a velocity based (andpercussive) instrument like a piano.

The device may function as a direct user interface to a real organ suchas a real pipe organ, or it may activate other synthesis devices by wayof Musical Instrument Digital Interface (MIDI) output, serial output,wireless control, and the like. Because the manifold 300 is made ofcopper, it can advantageously shield the system, and thus the fact thatcopper is a common plumbing material as well as the most commonelectrical conductor, is advantageous. Internally a loop antenna 313 canstill transmit through the copper since magnetic fields can thereoutwards propagate. Loop antennas, unlike dipole antennas, provideoperation despite the copper shielding which serves to keep electricalnoise out of the system.

Ordinarily, water from city water pressure mains is at much higherpressure than needed for the instrument. City water pressure istypically two to four atmospheres. One atmosphere is approximately equalto 10.3 meters of head, i.e. approximately equal to the maximum heightof head that people enter swimming baths from (e.g. municipal swimmingbaths that have towers with 10-meter platforms). Thus water pressure isapproximately two to four times higher than that experienced whilebathing in the most extreme way at a pool (i.e. approximately 50kilometers an hour impact with water after departing from the 10-meterplatform).

To convert from the approximately 20 to 40 meter head, down to thelesser pressures needed for the apparatus, a flow control valve, orpressure regulator may be used.

However, it is preferable to recover that energy and use the energy topower the instrument, realizing the sheer magnitude of this energy thatwould otherwise go to waste. Thus an energy recovery module 302 maypower the instrument.

Novice players may apply adhesive tape labels 312 to each such module,to label the notes. Alternatively liquid crystal displays in the modulesmay interactively display the notes as well as learning information forlessons, such as highlighting which note to play next.

The water jets may also be output devices either by illumination, color,or by tactile vibration, spray height variation, and the like. In apreferred embodiment, all of the jets are green when they are activeidle. To make the instrument easier to play, jets that are not used in aparticular song may be shut off. Alternatively, it is preferable to keepall the jets running for aesthetic value, but only illuminate the onesthat are part of a given song. For example, to play “Amazing Grace”(words, John Newton 1779, music, Carrell and Clayton, 1831) only six ofthe jets, namely C, D, F, G, A, and C, are needed. The others may beshut off, or their lights shut off, and a single green light may guidethe user through the song, to light up the jet that the user should hitnext.

In fact jets could go all the way around the whole circle, even in theback where it is difficult to reach, while only the front jets (easierto reach) would need to be used to play music. Alternatively, the spacenot used by jets at is used for indicia 303 such as trademarkinformation e.g. as shown “FROLICious FUNtain (TM)” along with usageinstructions, and the like.

Various modes such as teaching mode, and song to learn, are selected byholding down different combinations of jets at power up time. Unusedchord combinations are used as symbols to type messages into a computerto select processor 140 operation. Water typing modes are selectable totype in song names, search parameters, etc., but the water typing is notso bad as mid air typing. It is known that air typing is difficult, likeplaying air guitar, since there is no feedback but water typing (orwater guitar) are made easier by the feedback.

If learn mode is shut off, all jets glow green, until pressed down. Asthe hand enters the spray, the jet turns yellow, then orange, then red.This is by way of a 3-terminal LED that has red and green elements, andthe LED also forms part of the computer vision system that sees thewater spray flow diverted.

Thus the LED serves double duty as the light source for the visionsystem and the illumination. Since the illumination is nice and subtleit need not be visible to others, but can be if desired, by playing in adarkened room. In this way, teach mode can be hidden from others, sothat in a liquid interface karaoke setting, only the player can see theprompting.

Alternatively, the apparatus of FIG. 3 can be used as an interface toother equip ment, such as a computer. For example, the apparatus may beused by a disc jockey to play prerecorded music. By spinning the handaround in the circle of water jets, the virtual disk is spun to“scratch” or timewarp or modulate the music. Two such liquid userinterface rings, i.e. two manifolds 300 may be used to simulate twoturntables, to create a virtual mixing platform. Since many disc jockeysalready perform in their boxers or briefs, and since many of the danceclubs have a “foam party” or “beach party” theme (e.g. in many clubs theelectrical systems are already wet-safe) the apparatus of the inventionmay find many applications in such dance and performance orientedspaces.

The circular shape of the apparatus of FIG. 3 is by no means limiting.For example, the apparatus could assume other shapes such as that of thehollow fiberglass frogs commonly found in splash pads and spraygrounds.Alternatively, the apparatus may be built into a swim ring made to floatin the water, with the apparatus being entirely self-contained. In thiscase, the user can put expression into the music by dunking or partiallydunking the instrument while playing. The sound can thus be affected bythe way the instrument is sloshed around in the water.

A curved portion of pipe may be used, so that the floating andself-contained apparatus can be moved through the water. Other flowsensors can be installed in the instrument to measure how it is beingpushed through the water, and thus control the sound or flow of water inaccordance with movement through the water. For example, a “mouth” atone end of the instrument can be fitted with a flow sensor to allowthere to be an “embouchure metaphor” in which a user pushes theinstrument through the water and the water flowing into the mouth issensed, and this measures quantity controls the flow of water out thejets. In this way, the interface jets rise and fall in response to theamount of water “pushed” into the instrument's mouth. Thus the user canbelieve, or choose to believe, that the water coming out of the jets isdue to the water pushed into the mouth of the instrument. To the extentthat a pump may be controlled or modulated in this manner, an embouchurepower assist arises in which a user can appear to make the jets rise andfall by moveing the instrument faster or slower through the water. Byhaving these changes in speed of the instrument moving through the wateraffect the sound, the instrument thus becomes more expressive in a waythat is intuitive for a user to understand. This form of expressioncomes in addition to the move obvious dunking and lifting of theinstrument to change volume and tone. Various float and flow sensors canthus be used to make the embouchure of the instrument more richlyexpressive.

FIG. 4 illustrates the vacuum exclusion principle of the multijet systemof FIG. 3. Hand 130 descends to partially block one of jets 310, thusreducing the amount of water that comes out of that jet. The lower thehand 130 descends, the less water can come out of the jet 310 that isunder the hand. At least some of the water that would have come out thatjet goes out the other jets. Typically blocking one jet results inincreased flow out of the other jets. Additionally, each jet has a “T”fitting 400, so that when one jet is blocked water gushes out of theblocked T fitting side discharge 410. Note that “T” fitting 400 is not areducing “T” fitting, although it may be spliced in by way of anadditional reducing “T” fitting 499. Also it is important that jets 310,when not blocked deliberately by the user, do not offer significantlymore resistance to water flow than discharges 411.

Interestingly, no water comes out of the other side discharges 411. Infact, the more jets that are blocked, the faster the water gushes outtheir side discharges and out of the other jets, but at the same time,an even stronger vacuum is created on the unblocked side discharges 411.Thus initially, where all the side discharges are under slight vacuumwhen none of the jets are blocked, the unblocked side discharges 411 arepulled under even greater vacuum when more flow comes out the unblockedjets, either because other jets are blocked, or when water pressureincreases, or the like.

This system works very well, so long as the “T” fittings 400 are smallcompared with the size of the manifold 300. Various kinds of flowmeters, pressure meters, or the like, attached to discharges, will workquite well. In a preferred embodiment, the discharges point to thecenter, and a flow meter is used, because this allows the bather to getsplashed by the discharges, and thus receive tactile feedback. In thisway, blocking the jet with the finger or hand results in the bodygetting splashed by discharge. This often improves the ability of theplayer to become one with the machine of the instrument. A satisfactoryflow meter is a vision system that uses a discharge lens property. Lightsources 120 are blocked from shining into optical sensors 150 by baffles170. Each of the nine discharges has one baffle 170, one light source120, and one optical sensor 150. In this multijet embodiment, anindividual photoresistor is used for each discharge, rather than asingle camera. The circular array of nine photocells (photoresistors)may be thought of as a nine pixel camera if desired, from a conceptualpoint of view. When water flows out through discharge 410, the sprayforms a crude but sufficiently effective lens that light rays fromsource 120 reach sensor 150. Photocells of sensor 150 should pointdownward and the lights 120 should point up for 2 reasons:

-   -   ambient light tends to come from above, and thus downward facing        sensors 150 will be less adversely affected by the ambient light        that might otherwise result in false triggering of musical        notes;    -   light sources 120 have the advantage of being visible to the        user when they are facing upwards.

Obviously a cover may be used to shroud each of the lights, but it isnice to be able to operate the apparatus with the covers off to see whatis happening inside, or to use partially transparent covers forepistemological or experimental reasons or pure aesthetics. Of courseadditional lighting may be used in a playing on the instrument, and thisincludes lights on the instrument as well as elsewhere. For example,nine external MIDI controlled or computer controlled stage lights may beused, one for each note, so that a single solo performer may run anentire virtual band, and lighting console, while singing. A virtual bandmay be indexed through so that the user plays the lead role (whilesinging), on the apparatus of the invention. The entire band may beorchestrated by processor 140, such that all the instrumentsautomatically adjust in time with the lead music from the user.

A satisfactory photocell is a cadmium sulphide photoresistor such as thekind used in dusk to dawn electric eye lights. Such a photocell may beconnected directly into the matrix of most musical keyboards to activatea note, since flow results in light diverted to an otherwise baffledphotocell, and since light results in less resistance (moreconductance), which is like pressing a key on a keyboard.

The same is true of computer keyboards, so the apparatus can be directlyconnected for water typing or playing music with little or no interfacehardware or power supply needed by the input device itself, other thanfor light, which could, in principle, be just ambient light if thephotocells were moved down to the bottom. However, in preferredembodiments, for resistance to moisture effects, a lower impedancethreshold is desired, and for other reasons (e.g. more bits of amplitudecontrol) an active powered system is preferred. In some preferredembodiments, directional photodiodes or phototransistors are used forsensors 150. Typically a 7 bit precision is used to quantize the amountof flow, although greater precision and a lookup table are sometimesdesired, to shape the amplitude response of the instrumentcomparametrically. The displays for note labels 312 on each note arepreferably square computer displays, so they adapt well to a comparagrameditor, for setting the note's amplitude response.

In other embodiments, an additional vision sensor overall such as anoverhead camera for all nine jets or an additional sensor on each jet,or a different design is used to measure direction of water spillage,slappage, etc., so that the jets can be played more expressively. Forexample, to play along while singing the word “don't” in “hush littlebaby, DO-N'T you cry” or “standing” in “with mommy and daddy standingby” (Summertime, 1935), one presses down on the the high C jet andsweeps the water to the left, toward Bb, prior to laying into the Bbfrom the other direction. The result is the nice sounding down-chirpthat so expressively captures the closing words of the the lullaby.

FIG. 5 is a diagram depicting a multi-jet liquid user interface fullycontained inside a copper pipe manifold 300. This provides a very simpleaesthetic in which the instrument becomes a nice looking coppersculpture.

Sensors 550 are simply pressure switches rather than optical sensors. Anacceptable sensor is a miniature version of something found in a Reznorduct furnace for checking to make sure the air is flowing through theduct before the natural gas is switched on. Any switch in thesensitivity range from 1 to 20 inches of water column will work quitewell. The switches of sensors 550 and wiring 551 are inside the manifold300. Holes 510 in front of each port of sensors 550. Water pressuresupplied to the pipe forces water out all of the holes, creating avacuum on all of sensors which keeps them from activating, except when auser wishes to block one or more of the holes in which case positivepressure activates sensor 550 to produce a user input. Preferablysensors 550 are back vented by vents 511 so they can see atmospheric airpressure as a reference pressure.

The resulting embodiment with holes 510 can be played like a pennywhistle, tin flute, or other similar wind instrument, except that it isa water instrument interface played by blocking water from coming out ofcertain holes. In particular, the processor 140 can be programmed tooperate so that hole fingering is that of any preferred instrument ofthat type. Instead of the circular manifold 300, a straight manifold canbe used, and the different size holes of a penny whistle or tin flutecan be used, and thus preserve the familiar fingering of thatinstrument.

FIG. 6 shows how water may be diverted from main jets 610 to smallerside jets 611, so that a more immersive multimedia input device may thusbe created. For example chords may be activated with one finger, byblocking multiple jets at the same time. By skipping by twos, harmoniousgroupings are possible so that sloppy fingering results in good soundsthat harmonize well, in much the same way that a harmonica is designedso that sloppy playing results in good sound by blowing through adjacentholes to get harmonious sounds.

Thus in this embodiment of the invention, harmonious groupings of ofnozzle jets 611 facilitate easy chording.

FIG. 7 shows some examples of fingering positions for a popular song“What shall we do” (song of unknown authorship). Words to the song,chord suggestions, etc., or product information such as labeling (e.g.“Playing in the fountains (TM)” may be displayed in display field 730.

Multiple jets 611 can be simultaneously covered with one finger 710, tomake a D minor chord. Finger 710 is shown as a solid line. By movingover and down, the second row of nozzles can be activated in similargrouping to get a C-Major chord with finger 720. A drawback of thisdesign is that it is hard to get to the top row without affecting thebottom row slightly, especially when jets 611 shoot high.

Accordingly, preferably nozzle groups are brought closer together andrear ranged, so that fingertips can be inserted into the spray. Finger711 plays a D minor chord, and any three jets in which there is one jetclosest to the user plays minor. Any three jets with two toward the userplays major, such as the C Major of finger position 721. A display area740 prints the words to the song and shows fingering positions forteaching mode.

FIG. 8 shows an example of a very simple embodiment, more of anillustrative early embodiment than the preferred embodiment, since itshows some aspects of the invention in a way that is easy to understand.For simplicity (but not to suggest it is the preferred embodiment) theinput device of FIG. 5 is considered for sensor 550 shown. Ordinarily, anaturally open or naturally closed switch sensor 550 would bridge overand give erroneous results due to conductivity of treated water. Thus toattain immunity to water conductivity, sensor 550 is has both itsnaturally open (N.0.) contact 809 as well as its naturally closed (N.C.)contact 800 in use, in addition, of course, to its common (C.) contact805. These contacts are connected respectively to the ground 810, middle815, and tip 819 of a stereo ¼ inch plug 830 by wire cord 840. Cord 840is preferably a round flexible black wire in which the ground is ashield around the other two wires.

A number of jacks (sockets) are provided for the insertion of a numberof plugs 830. The number of plugs 830 is typically equal to the numberof jets which is typically 9 for a simple instrument that can be playedby children or inexperienced users, though more sensors may be used formore complicated pieces.

The 9 plugs 830 can be plugged into some of the sockets 850 on processor140 to select a key in which to play. For example, to play in C-Major,or D dorian minor, the nine plugs are inserted into the C, D, E, F, G,A, B, C, and D sockets 850. The 3-wire interface allows processor 140 todetect which notes are plugged in and to display this information suchas on LED 861, or alphanumeric or computer display 870. One way forprocessor 140 to display its knowledge of which sensors were plugged in,is for it to display some possible songs that can be played in the keyso selected.

Light source 861 will illuminate when the input line 865 is pulled lowby plug 830 connection from middle 815 to ground 810. This connectionwill also very decisively short the coil of relay 862, to definitelykeep it off.

At this point, blowing into port 820 of sensor 550 will test the systemand play a note, so even without the availability of water, theinstrument can be tested by blowing into the holes of each note. Blowinginto port 820 will cause contact 805 to disconnect from contact 800 andthen to connect to contact 809. This will lift the coil of relay 862 toenergize it, along with energization of the LED 860. Preferably thereare similarly two LEDs inside each jet 310, and preferably the LED 860that is on when the jet is blocked is red, and the LED 861 that is onwhen the jet is not blocked is green. When the jet changes color to red,it will still be visible through the flesh of the user, since flesh ismore red in color than green. Moreover, in transmission, flesh is veryred (i.e. it is somewhat translucent in the red). Thus the finger on theblocked jet 310 will be visible in red to affirm as a form of visualfeedback that the instrument is working and has responded. This isuseful when using a music synthesizer with slow attack, so that the usercan know exactly when a note is actuated, i.e. that the jet 310 has beenblocked or depressed enough to be considered a note-on, prior to evenhearing the note.

It is essential to have a break before make type of switch, to avoidshorting the power supply, but most pressure switches are of this type.Preferably the switch can be modified to remove hysteresis or deadband,so that it can function as a velocity sensing switch, so that processor140 could determine how fast the water jet is blocked, to adjust theamplitude of the musical note in response to how hard each jet 310 ishit. However, this feature is not shown in the simple relay embodimentin processor 140, in order to make the diagram simple. In implementationthis velocity is found. (calculations in processor 140) by computing thetime between break and make.

FIG. 9 depicts a platform 910 for immersive multimedia, with a fluiduser interface in which the user's entire body, not just his or herfingers, is used in the immersive multimedia input device. Thisembodiment of the invention may be installed at a municipal swimmingbath 900 where there is a tower, or at locations without a tower, sinceit is also possible for people to enter from a springboard, or even toenter just by jumping off the side of the pool (0-meter platform) or tointeract by frolicking in the pool, or with similar immersive multimediain a lake or ocean.

In the embodiment shown, the user 901 climbs the tower, up onto theplatform 910. A satisfactory platform is the standard 5-meter platform(for approximately 1 second in the air, and 36 km/hour speed of entryinto the water user interface) or 10-meter platform (for approximately1.4 seconds in the air, and 50 km/hour speed of entry into the wateruser interface) that may be found at many municipal swimming baths,university pools, and the like, as are often referred to as “olympicpools” (because entering a pool from a height of 10 meters is an olympicevent, as well as a form of recreation for children).

A fun and playful splash screen or splash page 200 is projected from alight source 120 hung from the bottom of the platform, as projected rays920 that span all or some of the pool area. Most platforms are cementwith railings made of structural pipe fittings with size 8 being themost common size of structural pipe fitting found on platform railings.Various fittings commonly used in the theatrical lighting industry maybe used to temporarily attach light source 120 to the bottom of theplatform with appropriate rigging, using Alvin pipe clamps (e.g. fromAlvin Industrial Sales in Canada). Standard safety procedures forrigging are used, i.e. safety chains on the light source in case itworks loose over the years when a temporary one-day installation mightbe kept for 10 or 20 years in change of mind. A satisfactory lightsource for this embodiment of the invention is a high power dataprojector, or a laser based vector graphics projector. The projectorprojects streaming media such as a scrolling sign of splash page 200,with the words rolling down the musical scale, so that user 901 canselect a key in which to later play the music. The user 901 selects thekey by departing from the platform.

When departing from a platform, bathers typically insert their handsinto the pool 999 first, so that the water hits them from above. In thisway the user's body is upside down at time of impact, so that, allthings being relative, in human-centered coordinates, the water in thepool pours down on top of the user. This water-from-above results in anexperience similar to (though much more extreme) a shower, where waterfalls down on top of a person.

In this case, usually the hands 130 of user 901 will be the first bodypart to hit the water (hands are usually extended to cut through thewater to avoid getting hit on the head with the water).

Optical sensor 150 has a field of view that includes rays 950 to seesome or all of the pool 999 water surface and below, and is arranged todetect when and where the user's hand 130 hits the water. This point ofcontact selects from the splash page 200 in a manner similar to thatshown in FIG. 2, except that the wall of water or sheet of water of FIG.2 is now laid out flat and is the surface of the pool 999.

In addition to being a splash page for streaming media, the pool 999also functions as an immersive multimedia environment, because thesensor 150 can continue to observe user 901 in descent into the water,and the manner of entry can be used to select or affect options, or canbe otherwise used as a fluid user interface (i.e. as an input device) toa computer processor 140 or other input system or systems of theinvention.

The top of the platform 910 may also be used as a display surface 911,to display messages for the bather, such as cautionary notes if thesystem observed that bather in a suboptimal entry on a previous try, orto display emergency messages, since it is hard to hear lifeguards,etc., from way up on the platform. When surface 911 is not being usedfor emergency messaging, it may display fun product information such asshown, “The key to good music is to play in the water (TM)”.

In one application, this embodiment of the invention may be used to setthe key of another instrument in a water park, for example. Thus a usercan use the 10 meter platform as an input device to choose the key thata nearby fountain will then play in. This creates a fun and playful wayof having an input device for setting parameters for playing music.

Additional multimedia spaces include areas around the pool. For example,when the system detects the presence of a bather on the tower (i.e. onthe way up) or up on the platform, a cautionary message on deck surface922 may be projected to warn other bathers not to enter the pool at thattime. This feature may be added simply by extending the field ofcoverage of light source 120 so that it includes rays of light to ray921.

In addition to the splash screen of splash page 200, other playfulelements may be included. For example, a fish-based screen saver mayoperate at idle times, or interactive gaming elements 902 may bedisplayed on the bottom of the pool. Various games include “catch afish” in which a user needs to land on a fish, as well as “avoid thefish” in which a user needs to not land on a fish. The latter game ispreferable to the former, to teach bathers the safety skills of avoidingcollisions with other bathers.

Splash pages 200 may be projected on the bottom of the pool, or on thesurface, or simultaneously on both, or on various intermediate mid-waterareas, through the use of focus, and the like. A large apertureprojector can have limited depth of field, and when a black backgroundwith light colored lines is used, it can focus on the bottom withoutaffecting the surface, or vice versa. Two projectors, one for surface,and one for bottom can also be used together. Thus surface game elementssuch as element 903 may be combined with bottom game elements 902.

To enhance surface visibility the bubbler feature of most pools can beswitched on or modulated. Many swimming baths have bubble jets to reducethe severity of impact when bathers land poorly, and these bubbles couldbe modulated as projection surface. To make the immersive multimediainteractive, the bubble jets can be dynamic, with the splash pages 200.Also, if visibility of the bottom is desired, the bather can be tracked,and a burst of bubbles delivered just before the bather hits that water.

This results in loss of visibility of the bottom on the descent, butthis is not such a bad thing. Bathers generally learn that looking downinto the water, whether in head first or hands first entry, oftenresults in two black eyes and a badly bruised face. Thus it is evendesirable in the invention to blank out the splash screen 200 (such asby turning off light source 120) as soon as a bather departs from theplatform.

Where light source 120 is part of the vision system for sensor 150, andthe blanking feature is desired, the light may change to infrared orother invisible light source during the blanking interval, or for thewhole time so as not to rely on visible light.

The baths are a very social place, and particularly the towers, sincethe sequentiality of bathing is there mandated by safety, so thatbathers line up to use the platforms, and there is time for idle chatwhile standing in line. Additionally, the serialization of bathing(sequentiality) gives rise to a phenomena in which bathers are each ondisplay upon the elevated platform, one at a time.

The invention can thus be used for adding fun, games, music, or othermultimedia elements to such ritualized or social bathing.

The pool 999 need not be limited to a rectangular olympic style pool.For example, a round pool could be built, with an offset platform thathangs over it like the tone-arm on a record player. A projection of aspinning roulette wheel could then be the streaming media of splash page200, such that user 901 becomes the roulette ball. In this way, a personcould place a bet by entering the pool. Processor 140 determines thelanding time and place of the first part of the bather's body, and thespinning of the roulette wheel then would stop exactly when user 901 hitthe water. By continued display of a stationary roulette wheel, the userand others could wait in suspense until the water ripples from thesplash of the bather fade out, to reveal a clear image of where thebather hand landed on the virtual wheel. This adds the thrill of theplatform to the thrill of gambling, and turns a fun and silly game likeroulette into a fun and silly and splashy game.

FIG. 10 depicts the timing diagram for a two-jet embodiment of theinvention that can be used to both set and display state. Initially onlyone of the two jets is on. When the user pushes down on the jet that'son, the jet goes off, and the other jet goes on.

Pressing down on the active jet will cause it to turn off, and cause theother jet to come on. There are thus two states that the system can bein:

-   -   state 1: jet 1 is on;    -   state 2: jet 2 is on.

Thus the fluid medium can be used as both an input device, and an outputdevice. Input is obtained by pressing a jet. Output is obtained byfeeling the jets or, in the case of water, looking at the jets.

In the figure, the system is shown initially in state 1, i.e. jet 1 ison, and jet 2 is off. Timing diagram 1011 shows when the hole of jet 1is covered, e.g. when a user places his or her finger over the hole torestrict fluid flow out of the hole, if there is fluid flowing out ofthe hole. Timing diagram 1012 shows when the hole of jet 2 is covered.Timing diagram 1021 shows the output of a restrictometer measuring therestriction of flow from jet 1, whereas 1022 shows the output of arestrictometer measuring the restriction of flow from jet 2. In thiscase, each of these two restrictometers is made from the following twoitems:

-   -   a tee fitting; and    -   a pressure sensor attached to the side discharge of the tee        fitting.        fluid from each jet is passed through the straight portion of        each of the two tee fittings.

Timing diagram 1031 shows when pump 1 is turned on, to spray fluid outof jet 1, and timing diagram 1032 shows when pump 2 is turned on, tospray fluid out of jet 2.

The bold curved lines with arrows show the cause and effect relationshipof the timing diagrams apparatus of this embodiment of the invention.The bold curved solid lines with arrows show physical cause and effectrelationships, and the bold curved dashed lines with arrows showcomputational (virtual, i.e. induced by, for example, a microcontroller)cause and effect relationships.

Initially, on timing diagram 1012, when hole 2 is blocked, nothinghappens. This is because pump 2 is initially off. Blocking a jet thathas no flow going through it results in no change in restrictometerreading.

Initially restrictometer 1 shows a slight negative value because pump 1is running, and by way of the Bernoulli effect, a slight vacuum is drawnon the side discharge of the tee fitting.

Then, a little later, when hole 1 is blocked, sequence 1000 shows thatwhen hole 1 is blocked, after a short delay, restrictometer 1 goes to ahigh positive value. A direct process (such as a flow switch and relay)or a computational process (e.g. by way of a computer or microprocessorcontrol, microcontroller, or the like), is used in the invention tocause pump 1 to shut off, and to cause pump 2 to turn on.

A satisfactory microcontroller is the AVR manufactured by Atmel.Computational sequence 1001 shows that the signal is sent to pump 1 tocause it to go off after a slight delay. Computational sequence 1002shows that the signal is sent to pump 2 to cause it to come on after aslight delay.

Physical sequence 1003 shows that, shortly after the time when pump 1goes off, the restrictometric reading from restrictometer 1 falls tozero.

Physical sequence 1004 shows that, shortly after the time when pump 2comes on, the restrictometric reading from restrictometer 2 goesnegative, because of the Bernoulli effect on the side discharge of thetee fitting that now has fluid running through it.

Now we have a situation in which jet 1 is off, and jet 2 is on. Now, ifa user blocks jet 1, nothing happens, but when a user blocks jet 2,physical sequence 1005 shows that, after a short delay, an output fromrestrictometer 2 swings strongly positive. This positive swing is sensedcomputationally, and an on signal is sent to pump 1, as shown incomputational sequence 1006. Simultaneously an off signal is sent topump 2, as shown in computational sequence 1007.

When pump 1 comes on, restrictometer 1 swings slightly negative, asshown by physical sequence 1008. When pump 2 goes off, restrictometer 2falls to zero, as shown by physical sequence 1009.

This embodiment of the invention can be used by itself, for example, asa decorative on/off switch for a room light or table lamp. The tablelamp may even be built into a decorative fountain that has the two jetsin it, so that pressing down on a first jet 1 causes the the lights toturn on, and pressing down on jet 2 causes the lights to turn off.

If both jets are held down, this could cause the whole system to shutdown into a non-recoverable state. This may be desired as a way to shutoff the fountain, requiring it to be unplugged and plugged in again. Itwould, indeed, be a very intuitive way to shut the whole thing down.Alternatively, a special state could be entered in which both jets comeon together and stay on together.

As another use of this embodiment of the invention, instead of havingthe lights in the room mimic one of the jets (e.g. having the lights goon when jet 2 goes on), this embodiment of the invention can be usedwith other musical embodiments of the invention to select stops, like ina pipe organ. Pipe organ stops are normally pulled out to activate them,but here the stops are jets that are pressed in.

For example, a musical sculpture may have 61 jets for the 61 keys of afive octave fluid-based “keyboard”, along with two additional jets tothe left, that are used to select sound stops. Pressing down on jet 1might turn on a flute sound. Pressing down on jet 2 might turn on atrumpet sound. Thus the user can select from among flute or trumpet bypressing the two expression jets to the left of the 61 main jets of theinstrument.

Moreover, a third state can be created, such that pressing down bothjets at the same time causes both pumps to come on, so that both arerunning, giving a combined flute and trumpet mixture.

More than two jets can also be used for this purpose. For example, threeexpression jets may be used as follows: To initialize: turn on pumps 1and 2; while (1) // i.e. then enter an infinite loop as follows: if jet1 is restricted, then turn off jet 1 and turn on or keep on jets 2 and3; if jet 2 is restricted, then turn off jet 2 and turn on or keep onjets 1 and 3; if jet 3 is restricted, then turn off jet 3 and turn on orkeep on jets 1 and 2; end while

If desired, multiple jet presses can be detected, e.g. if jets 1 and 2are held down together, the program senses that more than one jet isrestricted, and the outcome is to cause only pump 3 to come on. If allthree are held down, then the device could be shut down (all pumps off)with some other way required to revive it, or a special excetion couldbe made and all pumps could come on, to prevent an irreversible statetransition.

Embodiments of the invention may also use proportional stops, e.g.pushing a stop down partway causes the pump to drop to some fraction offull flow, but not turn completely off either. In this way, the stopscan be adjusted up or down in varying degrees.

In another application of the invention, one or more jets at onelocation can control one or more jets somewhere else. For example,pushing down on the jet of a fountain in Toronto can cause the jet tostay down, while turning on a corresponding jet of a fountain inAustralia. Thus the apparatus of the invention creates the illusion of asolid rod of water passing through the earth, that, when pressed on oneend, comes out the other end of the earth, and vice versa.

FIG. 11 depicts an embodiment of the invention having 320 jets (16 by 20array of jets). This is enough jets to form a recognizable image inwater, such as the impression of a hand pressing down on the jets. The320 jets 1100 each consist of a hole drilled into a six inch (approx.150 mm) blue plastic schedule 40 watermain pipe. The pipe is bent in anice long arc, with 61 note jets on it, to play music, and the array of320 expression jets is to the left of the note jets, so that a handprint can be used to set the expression (timbre, and other qualities ofthe sound).

If all the holes are equal in size then there is a problem with the jetsthat bend around the pipe being of greater pressure than the high anddry jets at the top of the pipe, so it is preferable to make a hole sizeprofile, so that holes drilled at the top are a little larger than holesdrilled toward the edges. As pictured, the top row, R0, of holes and thebottom row, R15 of holes are further around the pipe, but the middlerows between are high on the pipe, and thus the middle row holes shouldbe made larger and the top and bottom row of holes smaller, and aprofile of hole size created to even out the flow of water out of theholes.

An underwater video camera 1105 is mounted inside the pipe looking up atthe holes. When a user puts his or her hand onto the array of holes, thecamera can “see” which holes are restricted. In this case the camerafunctions as an array of restrictometers, so that it measuresrestriction of the various jets, optically. The processor 140 analyzesthe video image from the underwater camera. The array pattern of thejets blocked can thus be used as musical expression to affect the sound.

In some embodiments of the invention, it may be desired for theexpression pad to function as both an input and an output device. Rowand column servos 1101 and 1102 serve the function of making theapparatus work as an output device.

Row servo 1101 drives row flapper wires 1103 which, in some embodimentsof the invention are conductive wires affecting magnetic core flappers,that work like magnetic core memory with column flapper wires, toaddress individual jets.

In other embodiments of the invention, flapper wires are stiff stainlesssteel wires that actuate mechanical hole blockers, to turn on and offindividual jets.

Column address flapper wires 1104 may curve around the pipe, and carrythis mechanical motion along a curved trajectory. Alternatively, a flatsurface may be used to avoid this matter.

The 320 jets are controlled as follows: For example, three expressionjets may be used as follows: To initialize: turn on all of the jets,then wait until some are restricted: while (1) // i.e. then enter aninfinite loop as follows: for (column=0:19) for (row=0:15) check regionof camera to determine which jets are restricted; if flow ofjet(row,column)=restricted, then engage flapper stop(row,column);end//for end//for end//whileHere, restricted means that the ambient light from the outside world isrestricted, so the restrictometer is measuring the flow of light.

If desired, the handprint can remain on the expression pad indefinitely,which has a very nice visual aesthetic, in which the very liquidmaterial of water can take on a permanent shape.

Alternatively a clearing function can be made that sustains the handprint only as long as there is no further restriction. Since there issome noise it would be preferable to set a restrictometric thresholdthat keeps the handprint there until someone starts to play with theexpression pad enough that more than ten percent of the remaining jetsare blocked, before the hand print is cleared.

Alternatively, a gradual dissolve can be applied that makes the handprint melt away after 2 or 3 minutes of inactivity.

In other embodiments of this invention, two expression pads can be usedto communicate or play. For example, pressing down on an expression padin Toronto might cause the hand print to appear in Australia on anotherexpression pad there. Thus one can imagine that water jets to be longglass rods that pass through the center of the earth.

From the foregoing description, it will thus be evident that the presentinvention provides a design for a wearable display or camera viewfinder.As various changes can be made in the above embodiments and operatingmethods without departing from the spirit or scope of the invention, itis intended that all matter contained in the above description or shownin the accompanying drawings should be interpreted as illustrative andnot in a limiting sense.

Variations or modifications to the design and construction of thisinvention, within the scope of the invention, may occur to those skilledin the art upon reviewing the disclosure herein. Such variations ormodifications, if within the spirit of this invention, are intended tobe encompassed within the scope of any claims to patent protectionissuing upon this invention.

1. A user interface, said user interface including: a housing, saidhousing having a plurality of openings, each of said openings for beingselectively blocked by a user of said user interface; a plurality ofrestrictometers, each of said restrictometers supplying an electricalsignal that is responsive to an extent to which a corresponding openingis obstructed by a user of said user interface; an output medium, saidoutput medium responsive to input from said plurality ofrestrictometers.
 2. A musical instrument based on the user-interface ofclaim 1, said user interface also including a fluid supply to saidhousing, said fluid emerging from said openings.
 3. A user interface,said user interface including: a fluid supply; a plurality of fluidjets; a plurality of sensors to each sense interaction between a user ofsaid user interface and said fluid; an output medium, said output mediumresponsive to input from said plurality of sensors.
 4. The userinterface of claim 3 in which said output medium includes a processor,said processor including a decision process, said decision processselecting from a plurality of symbols each in response to obstruction ofone of said fluid jets.
 5. An input device, said input device including:a body of water divided into a plurality of symbol areas; at least oneuser sensor for sensing interaction between a user and said body ofwater, said sensor for sensing which of said symbol areas a userinteracts with; a processor responsive to an input from said sensor,said processor for determining which of said symbol areas said user isinteracting with; an output made in response to the selection of saiddetermining which of said symbol areas said user is interacting with. 6.The device of claim 5 in which said interaction is touch.
 7. The deviceof claim 6 in which said body of water is an upward directed water jet,and said symbol areas are regions of height.
 8. The device of claim 7,further including a jet height controller, said jet height controllerresponsive to an input from said processor, an input signal to said jetheight controller being derived in response to which of said symbolareas are selected.
 9. The device of claim 7, further including atactile jet controller, said controller responsive to an input from saidprocessor, an input signal to said controller being derived in responseto movement between said symbol areas, said tactile jet controllerrapidly altering a user feelable aspect of said jet in response to themovement between said symbol areas.
 10. The device of claim 8, includingmeans for measuring location of said touch, said height being set equalto a height determined to be nearest said location, said heightmaintained at that level by way of a closed-loop controller.
 11. Amusical instrument based on the device of claim 7 where each of saidsymbol areas corresponds to a musical note, and said output is thesounding of a tone corresponding with said musical note.
 12. The deviceof claim 6 in which said body of water flows through a manifold having aplurality of jets, and said symbol areas are the jets.
 13. A musicalinstrument based on the device of claim 12 where each of said symbolareas corresponds to a musical note, and said output is the sounding ofa tone corresponding with said musical note.
 14. A water keyboard basedon the device of claim 12 where each of said symbol areas corresponds toa keyboard entry, and said output is the generation of a symbolcorresponding with said keyboard entry.
 15. The device of claim 14 inwhich said symbol is a discrete symbol with at least one additionalattribute.
 16. The device of claim 15 in which said attribute is thetime at which said symbol is selected by said user.
 17. The device ofclaim 15 in which said attribute is an attack time and a release time.18. The device of claim 15 in which said attribute is an amplitudeattribute, and in which said amplitude attribute is proportional to howfar down said jet was pressed.
 19. The device of claim 6 in which saidbody of water is a pool, said symbol areas being regions of the pool,said touch being by the first part of a body of a user entering saidpool.
 20. A musical fluid pipe organ flute comprising a housing; aplurality of holes in said housing; means for supplying fluid to each ofsaid holes; detection means to separately detect restriction of fluidflow emerging from each of said holes; sounding means for producing aunique sound in response to the restriction of each of said holes. 21.The musical fluid pipe organ flute of claim 20, further including anembouchure controller, said embouchure controller for affecting saidmeans for supplying fluid, in response to a mouth input of said musicalfluid pipe organ flute.
 22. A water-pipe-organ-flute comprising ahousing; a plurality of holes in said housing; means for supplyingliquid to each of said holes; a restrictometer associated with each ofsaid holes, said restrictometer for one of: detection of fluid flowblockage; or estimation of the degree of restriction, of liquid comingfrom each of said holes; sounding means for producing a unique note in amusical scale, in response to the restriction of liquid from each ofsaid plurality of holes.
 23. The musical fluid pipe organ flute of claim22, further including an embouchure controller, said embouchurecontroller for affecting said means for supplying liquid, in response toan input from a mouth of said musical water-pipe-organ-flute, said mouthof said musical water-pipe organ-flute for being at least partiallyimmersed in water, while said musical water-pipe-organ-flute is pushedthrough a body of water.
 24. A florgan, said florgan comprising: ahousing; a plurality of holes in said housing; means for supplying fluidto each of said holes; a restrictometric diverter for each of saidholes, said restrictometric diverter for diverting fluid to arestrictometeric discharge, when the corresponding hole is blocked; asounding means for producing a unique note in a musical scale, saidsounding means connected to said restrictometric discharge associatedwith each of said holes.
 25. A florgan, said florgan comprising: ahousing; an array of holes in said housing; means for supplying fluid toeach of said holes, said means including a side-discharge for each ofsaid holes, said side discharge arranged to: draw a vacuum when fluid isemerging from said hole; go into positive pressure when fluid isprevented from emerging from said hole; a sounding means connected toeach of said side-discharges, said sounding means producing sound whensupplied with one of: fluid flow; fluid pressure.
 26. The florgan ofclaim 25 in which said sounding means is an acoustic organ pipe.
 27. Theflorgan of claim 25 in which said sounding means is a device that makessound when water flows through it.
 28. The florgan of claim 25 in whichsaid fluid is water, and said sounding means comprises a steam boilerand steam whistle combination, such that said sounding means producessound by converting said water into steam, said steam entering saidsteam whistle, said florgan including an array of steam whistles to forma musical scale.
 29. The florgan of claim 25 in which said soundingmeans comprises a sound synthesizer connected to a processor, saidprocessor responsive to at least one input from at least one: fluid flowsensor sensing said fluid flow; fluid pressure sensor sensing said fluidpressure.
 30. The florgan of claim 25 in which said sounding meanscomprises activation or volume change of at least one note of a soundsynthesizer connected to a processor, said processor responsive to aplurality of inputs, each of said inputs coming from one of: a fluidflow sensor, sensing said fluid flow; or a fluid pressure sensor,sensing said fluid pressure.
 31. A florgan, said florgan comprising: ahousing; an array of holes in said housing; means for supplying fluid toeach of said holes; a sensor for each hole, said sensor for sensing atleast one of: proximity of an object or body part of a person; motion ofan object or body part of a person, irrespective of flow of said fluid;means for creating an output sound in response to proximity or motion ofan object or part of a person's body, to each of said holes.